Hotel ranking

Background

OTA Business Nature

OTA Company has two business models :

• Merchant Model

The company own what they sell. They have some flexibility in changing price. Their Revenues mostly from profit margin.

• Agency Model

More like broker. Revenue from commision and fees.

Some of OTA Company which apply Merchant Model : - KAI - AirAsia - etc

Some of OTA Company which apply Agency Model : - AirBnB - Booking.com - Expedia - Traveloka - Tiket.com

Let say there is OTA company , you can name T.LLC The problem is that : - The Conversion has dropped 10% from last month

As data scientist we should help business teams to fix the issue

After some analysis for example session replay + interview with product manager they said the customer does not convert when the recommendation show

There are several touchpoint of consumer conversion : 1. Home Feed Recommendation 2. Search Feature

To provide which one to prioritize we should check the data where most of the conversion take place. Via Home Feed Recommendation or Search Feature.

There is distinction between homefeed and search result. People which go to the search bar has already have intent to find hotel. Hypothetically the coversion should be bigger in search.

Hence solving bigger portion cover bigger problem.

Objectives

Our objectives is to build ranking system / services that satisfy both business and ml objectives

Business Objective

After all this ML Approach is initiative that aim to solve / optimize business metrics.

What type of metrics we are going to optimize ?

We need to know what is our business models. Its commision based model.

• If number of bookings increase → company income increase

• So number of bookings right ? Why dont use conversion.

Of course we can. But we are not using conversion because it can be mislead if we dont mention the numerator

Metrics : Total Bookings Increase by 20%

\(\textbf{Total Booking} = \text{Number of bookings from search activity}\)

ML Objective

Its about what you expect from your ML System.

Model Wise : Model Performance : 1. Can Beat Baseline Model in terms of NDCG (Previously 0.30)

Model Serving : 1. Max Latency 500ms - Search Experience is time sensitive - A company like google already estimated that they apps latency increase by 100ms it affects to their revenue

User Flow

It's Important to Understand User Flow

https://mermaid.live/edit#pako:eNpVkE1zgjAQQP9KZk84g5YPLZhDZxRsD7202lOHSwZWyACJDaHWiv-9AbW2Oexks-_t7uQIqcwQKGwruU8LpjR5ixNBzFlYUdtoWaMiK6FNfMbDXqpsRMbjB7K0FllGXltUB_LCFKvRIKOzuRyI6OZvUGgTmEoLssaPFht9QaMBjW_oGlPkn3il78iaiZKL3BSatrpq8aB1fEt4v5rpZ5BuZUUVT8v_DDGQkPovSLpHa1Pynem-SZWsqouxuhk102Z692QtpTw3BBtyxTOgWrVog9m1Zn0Kx76agC6wxgSouWZMlQkk4mScHRPvUtZXTck2L4BuWdWYrN1lTGPMWW4-8PdVochQRbIVGug08IcmQI_wBdQPg4kTuKHn9tENAhsOQN3Qn_iee3_veI4X-k5wsuF7mOpMpvPQn3uzmef6gR9M56cfOJiZCQ

System Design

https://mermaid.live/edit#pako:eNpVj81ugzAQhF_F2jNBpjYQfKjUQNVTD42aSyEHKyw_KtiRMW1TxLvXEDVS9rTjb2ZWnuCkSwQBVae_T400lrxnhSJunvLDgIa8jWguR7LZPJJdvgpyUCWawUpVtqomr66gO14zu9WX5nu0psUv2ZEXVGikbbW6M6arMcv3Un26I1dErixb2bMrIXd4oeBBbdoShDUjetCj6eUiYVpoAbbBHgsQbi2xkmNnCyjU7GJnqT607v-TRo91A6KS3eDUeC6lxayVtZH97dXg8tFUj8qCCJJkLQExwQ8IFkZ-HDDKaECjB8pY6MHFucKtzymPopBGMec8iGcPfte71OcJSwIebxmlNOER5fMfcAh0pg

In Eugene Yan's Blog Post Some Common System Design for Search and Recommendation could be decomposed into two components :

• Retrieval / Candidate Generation

Job : Filtering Out many items from 10.000 items to 1.000 items for example

• Ranking Stage

Job : Ordering the rank , this is the heavy duty job.

Does all the the system design for search and recommender system should contain both ? No. Its depend on number of item you have in catalogue.

The concern on this article is to highlight on Ranker Model Part

Model Development

Problem Formulation

Given User Query give ordered list of hotels that maximized chance user will book the hotel

Solution 1 : No Model Approach

The first stage is no ML at all. because if we just launch a product / company we dont have enough data, hence we can use simple heuristic such as : 1. Sort by Name 2. Sort by Price 3. Ranking Randomly

The goal is to obtain enough data to collect data to move on to the next step

Solution 2 : Modelling Approach

Our end task is ranking or to create ordering.

Now let's do some first principle breakdown :

How can we create ranking :

1. Predict the score of each item
2. Then Order the item based on the score

To produce the score is straight forward, to produce the ordering we want each item to be different in score not in discrete.

For example we could use predicted probability or ranking as starter.

Source : LucidWorks

Data Definition

A Glimpse of Data

So in ml based solution it requires data. What type of Information we show. Air BnB in their Article can classify the feature as : 1. Hotels / Listing Feature 2. User / Personalized Features 3. Query Features

Here is the idea :

1. If we use Hotel Features Only → it can produce ranking but we cannot expect variation in user level. If we compare recommendation of two users. Say A and B the recommendation would be the same.
2. To add more personalization we can add feature that unique to user level
3. To what extend its unique, if user characteristics does not change then the recommendation would be the same. However that is not ideal because user intention may be different at different search session

### Debiasing Features | Variable Name | Data Type | Description | Example | Notes | |-----------------------------|----------------|-------------------------------------------------------------------------------------------------------------|-----------------------------------------|-------------------------------------------------------------| | **search_result_rank** | Integer | Listing Rank In Search Result | `1` | | ### Listing / Hotel Features - It could be feature that describe facility , for example : - Number of rooms available between dates - Price - Location - etc - For now there are lot of companies using embeddings to express the feature of an entity - Usually embeddings are resulted from pretrained model (neural nets for example) | Variable Name | Data Type | Description | Example | Notes | |-----------------------------|----------------|-------------------------------------------------------------------------------------------------------------|-----------------------------------------|-------------------------------------------------------------| | **listing_id** | Integer | Unique identifier for each listing/experience. | 123 | Primary key for listing features. | | **price** | Float | Price of the listing in USD. | 150.75 | Range typically between 50 and 500. | | **rating** | Float | Average rating of the listing (scale 1–5). | 4.2 | | | **num_reviews** | Float | Number of reviews for the listing. | 45.0 | Generated using an exponential distribution. | | **bookings_last_30_days** | Integer | Number of bookings for the listing in the last 30 days. | 8 | | | **clicks_30d** | Integer | Number of clicks for the listing in the last 30 days. | 60 | | | **clicks_7d** | Integer | Number of clicks for the listing in the last 7 days. | 15 | | | **max_adults** | Integer | Maximum number of adults the listing can accommodate. | 4 | | | **experience_duration** | Float | Duration of the experience in hours. | 2.5 | | | **occupancy_rate** | Float | Occupancy rate of the listing (fraction between 0.5 and 1.0). | 0.75 | ### User Personal Features More Like User Preferences (Could be Implicit or Explicit), Past Bookings(If Available) for examples : - Past Bookings (1w,1month,3month,etc) - Average Stay - etc Okay, you might say what would happen if user is new ? AirBnB in their paper build separate model for login user and logout user. We could categorize new user as logout user due to the fact that the data is not available enough | Variable Name | Data Type | Description | Example | Notes | |-----------------------------|----------------|-------------------------------------------------------------------------------------------------------------|-----------------------------------------|-------------------------------------------------------------| | **user_id** | Integer | Unique identifier for each user. | 1, 2, 3 | Primary key for user features. | | **age** | Integer | Age of the user in years. | 25 | | | **gender** | Integer | Gender of the user (0 for male, 1 for female). | 0 | | | **user_clicks_30d** | Integer | Number of clicks made by the user in the last 30 days. | 18 | | | **user_clicks_7d** | Integer | Number of clicks made by the user in the last 7 days. | 5 | | | **user_booking_30d** | Integer | Number of bookings made by the user in the last 30 days. | 2 | | | **user_booking_7d** | Integer | Number of bookings made by the user in the last 7 days. | 0 | | | **user_avg_stay_30d** | Float | Average length of stays (in days) for the user over the last 30 days. | 3.5 | | | **user_avg_stay_6month** | Float | Average length of stays (in days) for the user over the last 6 months. | 7.2 | | ### Query Features What does Query Features Mean ? https://medium.com/airbnb-engineering/embedding-based-retrieval-for-airbnb-search-aabebfc85839 Features that describe criteria for hotels we are looking for , for example : - Number of guests - Number of Stay - etc The same goes with expedia in their article https://medium.com/expedia-group-tech/learning-to-rank-at-expedia-group-how-to-adapt-the-property-search-result-page-based-on-f4ebef78c94b which include : - checkin_date - checkout_date - We could also represent this as embedding | Variable Name | Data Type | Description | Example | Notes | |-----------------------------|----------------|-------------------------------------------------------------------------------------------------------------|-----------------------------------------|-------------------------------------------------------------| | **query_id** | Integer | Unique identifier for each query. | 42 | Primary key for query features. | | **checkin_date** | Datetime | Check-in date specified in the query. | 2024-03-15 | | | **checkout_date** | Datetime | Check-out date specified in the query. | 2024-03-20 | | | **lead_time** | Integer | Number of days from the query date to the check-in date. | 30 | | | **duration_of_stay** | Integer | Duration of the stay in days (difference between checkin and checkout dates). | 5 | | | **number_of_guests** | Integer | Number of guests specified in the query. | 2 | | ### Click Log Last but not least we need the data from search activity the purpose is to train model to maximize the chance of booking happened. The search process happen in whats called as **Session**. - Each User can have multiple session | Variable Name | Data Type | Description | Example | Notes | |-----------------------------|----------------|-------------------------------------------------------------------------------------------------------------|-----------------------------------------|-------------------------------------------------------------| | **session_id** | String | Unique identifier for the session (one session per user-query pair, formatted as "userID_queryID"). | "15_42" | | | **event_type** | String | Type of event: 'search', 'scroll', 'click', or 'book'. | "click" | | | **event_time** | Datetime | Timestamp indicating when the event occurred. | 2023-05-10T08:15:00 | | | **listing_id (in events)** | Integer/Null | For events associated with a listing (scroll, click, book); null for search events. | 123 (or null for search events) | ## Load Data

import pandas as pd 
import numpy as np 

# load click log 
click_log_data = pd.read_csv('dataset/click_log.csv')
click_log_data.head()

	user_id	query_id	session_id	event_type	event_time	listing_id
0	1	640	1_640	click	2022-09-05 11:30:37	736
1	1	640	1_640	scroll	2022-09-05 11:30:55	1644
2	1	640	1_640	scroll	2022-09-05 11:31:08	4963
3	1	640	1_640	scroll	2022-09-05 11:31:26	1909
4	1	640	1_640	scroll	2022-09-05 11:31:34	3779